Skip to content

Add benchmarks that run on the CI - #164

Open
Vipul-Cariappa wants to merge 3 commits into
compiler-research:mainfrom
Vipul-Cariappa:bench/function-calls
Open

Vipul-Cariappa wants to merge 3 commits into
compiler-research:mainfrom
Vipul-Cariappa:bench/function-calls

Conversation

@Vipul-Cariappa

@Vipul-Cariappa Vipul-Cariappa commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Look at the changelog file for a detailed explanation.
We run the benchmarks under callgrind and count instructions. We publish the results on GitHub Pages. A PR is compared against main's benchmark results and is considered a failure if it is more than 5% slower. We may want to adjust things depending on what we see in our CI later. We may want to extend this with cache misses later.
The current benchmarks are focused on templated and non-templated function calls with and without multiple overloads.

Once merged, we will need to turn on GitHub Pages on this repo.

@Vipul-Cariappa

Copy link
Copy Markdown
Collaborator Author

If you look at the Benchmark CI run's summary (https://github.com/compiler-research/cppjit/actions/runs/39566226517?pr=164), it provides a report on the benchmarks comparing against main branch:


Benchmark Report

✅ No benchmark runs more than 5% more instructions than main.
Comparing base (92525ea) to head (68803bd), in instructions counted by callgrind.
Counted main on this runner: it has no published counts.
History of main: benchmark charts.

@@                Benchmark Diff (geomean)                @@
##                                main       #164     +/- ##
============================================================
  first/callback                837.2 M    839.3 M    +0.25%
  first/class                   9.275 M     9.37 M    +1.02%
  first/overload                9.272 M    9.388 M    +1.24%
  first/primitive               8.547 M    8.638 M    +1.07%
  first/string                  22.06 M    22.22 M    +0.75%
  first/template                11.17 M    11.26 M    +0.80%
  first/template_overload       13.59 M    13.72 M    +0.97%
  hot/callback                  9.056 k    9.088 k    +0.36%
  hot/class                     2.869 k     2.87 k    +0.06%
  hot/overload                  2.826 k    2.826 k    +0.00%
  hot/primitive                 2.151 k     2.15 k    -0.05%
  hot/string                    9.877 k    9.874 k    -0.03%
  hot/template                  4.819 k    4.825 k    +0.12%
  hot/template_overload         2.904 k    2.904 k    +0.00%
============================================================
- Slower                                                   0
+ Faster                                                   0
  Unchanged                                              222
All benchmarks
Benchmark main #164 Δ
first/callback/invoke(double(*)(double)) 1.415 G 1.419 G +0.31%
first/callback/invoke(int(*)(const Point&)) 948.5 M 951.1 M +0.27%
first/callback/invoke(int(*)(int)) 478.3 M 480.4 M +0.43%
first/callback/invoke(std::function<int(Widget&,Color)>) 493.7 M 495.8 M +0.42%
first/callback/invoke[double](T(*)(T,T)) 1.043 G 1.043 G +0.00%
first/callback/invoke[int](T(*)(T,T)) 1.042 G 1.042 G +0.04%
first/class/pass_color 8.573 M 8.672 M +1.15%
first/class/pass_mixed_args 10.29 M 10.41 M +1.13%
first/class/pass_point 9.039 M 9.102 M +0.70%
first/class/pass_ptr_args 9.41 M 9.525 M +1.23%
first/class/pass_ref_args 9.489 M 9.549 M +0.63%
first/class/pass_shape 8.562 M 8.641 M +0.93%
first/class/pass_value_args 9.687 M 9.821 M +1.39%
first/overload/overloaded_1 9.271 M 9.398 M +1.36%
first/overload/overloaded_2 9.037 M 9.157 M +1.33%
first/overload/overloaded_3 9.543 M 9.658 M +1.21%
first/overload/overloaded_4 9.429 M 9.518 M +0.94%
first/overload/overloaded_5 9.091 M 9.215 M +1.37%
first/primitive/pass_bool 8.589 M 8.669 M +0.94%
first/primitive/pass_char16 8.498 M 8.572 M +0.87%
first/primitive/pass_char32 8.529 M 8.607 M +0.92%
first/primitive/pass_char 8.488 M 8.556 M +0.80%
first/primitive/pass_cref_bool 8.554 M 8.652 M +1.14%
first/primitive/pass_cref_char 8.519 M 8.609 M +1.05%
first/primitive/pass_cref_double 8.563 M 8.581 M +0.21%
first/primitive/pass_cref_float 8.498 M 8.589 M +1.07%
first/primitive/pass_cref_int 8.565 M 8.662 M +1.14%
first/primitive/pass_cref_ldouble 8.771 M 8.809 M +0.43%
first/primitive/pass_cref_llong 8.574 M 8.661 M +1.02%
first/primitive/pass_cref_long 8.594 M 8.653 M +0.68%
first/primitive/pass_cref_schar 8.511 M 8.615 M +1.23%
first/primitive/pass_cref_short 8.524 M 8.638 M +1.34%
first/primitive/pass_cref_uchar 8.504 M 8.603 M +1.17%
first/primitive/pass_cref_uint 8.619 M 8.726 M +1.24%
first/primitive/pass_cref_ullong 8.543 M 8.739 M +2.30%
first/primitive/pass_cref_ulong 8.553 M 8.622 M +0.81%
first/primitive/pass_cref_ushort 8.539 M 8.612 M +0.86%
first/primitive/pass_double 8.41 M 8.5 M +1.08%
first/primitive/pass_float 8.445 M 8.531 M +1.02%
first/primitive/pass_int 8.513 M 8.604 M +1.07%
first/primitive/pass_ldouble 9.018 M 9.123 M +1.17%
first/primitive/pass_llong 8.542 M 8.612 M +0.82%
first/primitive/pass_long 8.541 M 8.631 M +1.05%
first/primitive/pass_ptr_bool 8.559 M 8.629 M +0.82%
first/primitive/pass_ptr_char16 8.54 M 8.642 M +1.20%
first/primitive/pass_ptr_char32 8.581 M 8.645 M +0.74%
first/primitive/pass_ptr_char 8.527 M 8.643 M +1.36%
first/primitive/pass_ptr_double 8.493 M 8.59 M +1.14%
first/primitive/pass_ptr_float 8.513 M 8.61 M +1.14%
first/primitive/pass_ptr_int 8.551 M 8.621 M +0.83%
first/primitive/pass_ptr_ldouble 8.767 M 8.881 M +1.30%
first/primitive/pass_ptr_llong 8.592 M 8.628 M +0.42%
first/primitive/pass_ptr_long 8.618 M 8.661 M +0.49%
first/primitive/pass_ptr_schar 8.479 M 8.627 M +1.75%
first/primitive/pass_ptr_short 8.526 M 8.606 M +0.94%
first/primitive/pass_ptr_uchar 8.527 M 8.652 M +1.46%
first/primitive/pass_ptr_uint 8.514 M 8.678 M +1.92%
first/primitive/pass_ptr_ullong 8.561 M 8.653 M +1.07%
first/primitive/pass_ptr_ulong 8.54 M 8.624 M +0.98%
first/primitive/pass_ptr_ushort 8.53 M 8.62 M +1.06%
first/primitive/pass_ptr_wchar 8.573 M 8.697 M +1.45%
first/primitive/pass_ref_bool 8.542 M 8.652 M +1.28%
first/primitive/pass_ref_char16 8.528 M 8.613 M +0.99%
first/primitive/pass_ref_char32 8.533 M 8.638 M +1.23%
first/primitive/pass_ref_char 8.562 M 8.636 M +0.87%
first/primitive/pass_ref_double 8.493 M 8.598 M +1.23%
first/primitive/pass_ref_float 8.563 M 8.614 M +0.60%
first/primitive/pass_ref_int 8.551 M 8.627 M +0.89%
first/primitive/pass_ref_ldouble 8.731 M 8.817 M +0.99%
first/primitive/pass_ref_llong 8.539 M 8.615 M +0.89%
first/primitive/pass_ref_long 8.56 M 8.632 M +0.84%
first/primitive/pass_ref_schar 8.513 M 8.557 M +0.52%
first/primitive/pass_ref_short 8.516 M 8.653 M +1.61%
first/primitive/pass_ref_uchar 8.497 M 8.615 M +1.38%
first/primitive/pass_ref_uint 8.516 M 8.593 M +0.90%
first/primitive/pass_ref_ullong 8.559 M 8.633 M +0.86%
first/primitive/pass_ref_ulong 8.539 M 8.618 M +0.93%
first/primitive/pass_ref_ushort 8.508 M 8.594 M +1.01%
first/primitive/pass_ref_wchar 8.519 M 8.608 M +1.05%
first/primitive/pass_schar 8.472 M 8.547 M +0.88%
first/primitive/pass_short 8.517 M 8.597 M +0.94%
first/primitive/pass_uchar 8.463 M 8.577 M +1.34%
first/primitive/pass_uint 8.499 M 8.681 M +2.14%
first/primitive/pass_ullong 8.53 M 8.6 M +0.83%
first/primitive/pass_ulong 8.566 M 8.673 M +1.25%
first/primitive/pass_ushort 8.484 M 8.572 M +1.04%
first/primitive/pass_wchar 8.484 M 8.631 M +1.72%
first/string/pass_cstring 8.544 M 8.686 M +1.67%
first/string/pass_string 85.9 M 86.16 M +0.30%
first/string/pass_string_cref 34.36 M 34.48 M +0.34%
first/string/pass_string_view 9.389 M 9.456 M +0.71%
first/template/tmpl_deduce(bool) 10.81 M 10.93 M +1.11%
first/template/tmpl_deduce(float) 10.42 M 10.56 M +1.32%
first/template/tmpl_deduce(int) 10.52 M 10.6 M +0.75%
first/template/tmpl_pass[bool] 10.79 M 10.89 M +1.00%
first/template/tmpl_pass[char] 12.43 M 12.5 M +0.57%
first/template/tmpl_pass[double] 10.4 M 10.5 M +0.95%
first/template/tmpl_pass[float] 10.41 M 10.5 M +0.92%
first/template/tmpl_pass[long double] 12.93 M 13.03 M +0.72%
first/template/tmpl_pass[long] 10.6 M 10.63 M +0.28%
first/template/tmpl_pass[short] 10.61 M 10.67 M +0.55%
first/template/tmpl_pass[unsigned int] 12.24 M 12.32 M +0.68%
first/template/tmpl_pass[unsigned long long] 12.3 M 12.39 M +0.77%
first/template_overload/tmpl_overloaded_1(Particle) 12.6 M 12.66 M +0.46%
first/template_overload/tmpl_overloaded_1(Point) 12.3 M 12.44 M +1.09%
first/template_overload/tmpl_overloaded_1(Widget) 12.45 M 12.6 M +1.19%
first/template_overload/tmpl_overloaded_2(Particle) 14.78 M 14.95 M +1.14%
first/template_overload/tmpl_overloaded_2(Point) 14.58 M 14.82 M +1.65%
first/template_overload/tmpl_overloaded_2(Widget) 14.75 M 14.81 M +0.41%
first/template_overload/tmpl_overloaded_3(Particle,Point) 13.83 M 13.96 M +0.92%
first/template_overload/tmpl_overloaded_3(Point,Widget) 13.71 M 13.84 M +0.94%
hot/callback/invoke(double(*)(double)) 6.383 k 6.389 k +0.09%
hot/callback/invoke(int(*)(const Point&)) 8.14 k 8.146 k +0.07%
hot/callback/invoke(int(*)(int)) 6.191 k 6.197 k +0.10%
hot/callback/invoke(std::function<int(Widget&,Color)>) 14.28 k 14.3 k +0.19%
hot/callback/invoke[double](T(*)(T,T)) 11.29 k 11.46 k +1.47%
hot/callback/invoke[int](T(*)(T,T)) 10.63 k 10.66 k +0.24%
hot/class/pass_color 2.015 k 2.015 k +0.00%
hot/class/pass_mixed_args 3.717 k 3.717 k +0.00%
hot/class/pass_point 5.286 k 5.309 k +0.44%
hot/class/pass_ptr_args 2.23 k 2.23 k +0.00%
hot/class/pass_ref_args 3.454 k 3.454 k +0.00%
hot/class/pass_shape 1.988 k 1.988 k +0.00%
hot/class/pass_value_args 2.637 k 2.637 k +0.00%
hot/overload/overloaded_1 2.921 k 2.921 k +0.00%
hot/overload/overloaded_2 2.276 k 2.276 k +0.00%
hot/overload/overloaded_3 3.152 k 3.152 k +0.00%
hot/overload/overloaded_4 3.839 k 3.839 k +0.00%
hot/overload/overloaded_5 2.241 k 2.241 k +0.00%
hot/primitive/pass_bool 1.976 k 1.976 k +0.00%
hot/primitive/pass_char16 2.805 k 2.805 k +0.00%
hot/primitive/pass_char32 2.792 k 2.792 k +0.00%
hot/primitive/pass_char 2.718 k 2.71 k -0.29%
hot/primitive/pass_cref_bool 1.961 k 1.961 k +0.00%
hot/primitive/pass_cref_char 2.723 k 2.715 k -0.29%
hot/primitive/pass_cref_double 2.003 k 2.003 k +0.00%
hot/primitive/pass_cref_float 2.007 k 2.007 k +0.00%
hot/primitive/pass_cref_int 1.937 k 1.937 k +0.00%
hot/primitive/pass_cref_ldouble 2.008 k 2.008 k +0.00%
hot/primitive/pass_cref_llong 1.94 k 1.94 k +0.00%
hot/primitive/pass_cref_long 1.932 k 1.932 k +0.00%
hot/primitive/pass_cref_schar 2.76 k 2.752 k -0.29%
hot/primitive/pass_cref_short 1.93 k 1.93 k +0.00%
hot/primitive/pass_cref_uchar 2.764 k 2.756 k -0.29%
hot/primitive/pass_cref_uint 1.971 k 1.971 k +0.00%
hot/primitive/pass_cref_ullong 2.012 k 2.012 k +0.00%
hot/primitive/pass_cref_ulong 1.971 k 1.971 k +0.00%
hot/primitive/pass_cref_ushort 1.935 k 1.935 k +0.00%
hot/primitive/pass_double 2.013 k 2.013 k +0.00%
hot/primitive/pass_float 2.017 k 2.017 k +0.00%
hot/primitive/pass_int 1.957 k 1.957 k +0.00%
hot/primitive/pass_ldouble 2.02 k 2.02 k +0.00%
hot/primitive/pass_llong 1.978 k 1.978 k +0.00%
hot/primitive/pass_long 1.952 k 1.952 k +0.00%
hot/primitive/pass_ptr_bool 1.936 k 1.936 k +0.00%
hot/primitive/pass_ptr_char16 2.993 k 2.993 k +0.00%
hot/primitive/pass_ptr_char32 2.972 k 2.972 k +0.00%
hot/primitive/pass_ptr_char 2.817 k 2.809 k -0.28%
hot/primitive/pass_ptr_double 2.046 k 2.046 k +0.00%
hot/primitive/pass_ptr_float 2.049 k 2.049 k +0.00%
hot/primitive/pass_ptr_int 1.971 k 1.971 k +0.00%
hot/primitive/pass_ptr_ldouble 2.049 k 2.049 k +0.00%
hot/primitive/pass_ptr_llong 1.971 k 1.971 k +0.00%
hot/primitive/pass_ptr_long 1.971 k 1.971 k +0.00%
hot/primitive/pass_ptr_schar 2.817 k 2.809 k -0.28%
hot/primitive/pass_ptr_short 1.965 k 1.965 k +0.00%
hot/primitive/pass_ptr_uchar 2.762 k 2.754 k -0.29%
hot/primitive/pass_ptr_uint 1.97 k 1.97 k +0.00%
hot/primitive/pass_ptr_ullong 1.97 k 1.97 k +0.00%
hot/primitive/pass_ptr_ulong 1.97 k 1.97 k +0.00%
hot/primitive/pass_ptr_ushort 1.971 k 1.971 k +0.00%
hot/primitive/pass_ptr_wchar 2.139 k 2.139 k +0.00%
hot/primitive/pass_ref_bool 1.896 k 1.896 k +0.00%
hot/primitive/pass_ref_char16 2.447 k 2.447 k +0.00%
hot/primitive/pass_ref_char32 2.433 k 2.433 k +0.00%
hot/primitive/pass_ref_char 2.717 k 2.709 k -0.29%
hot/primitive/pass_ref_double 2.002 k 2.002 k +0.00%
hot/primitive/pass_ref_float 2.007 k 2.007 k +0.00%
hot/primitive/pass_ref_int 1.927 k 1.927 k +0.00%
hot/primitive/pass_ref_ldouble 2.007 k 2.007 k +0.00%
hot/primitive/pass_ref_llong 1.929 k 1.929 k +0.00%
hot/primitive/pass_ref_long 1.929 k 1.929 k +0.00%
hot/primitive/pass_ref_schar 2.717 k 2.709 k -0.29%
hot/primitive/pass_ref_short 1.923 k 1.923 k +0.00%
hot/primitive/pass_ref_uchar 2.722 k 2.714 k -0.29%
hot/primitive/pass_ref_uint 1.928 k 1.928 k +0.00%
hot/primitive/pass_ref_ullong 1.928 k 1.928 k +0.00%
hot/primitive/pass_ref_ulong 1.928 k 1.928 k +0.00%
hot/primitive/pass_ref_ushort 1.927 k 1.927 k +0.00%
hot/primitive/pass_ref_wchar 1.942 k 1.942 k +0.00%
hot/primitive/pass_schar 2.76 k 2.752 k -0.29%
hot/primitive/pass_short 1.95 k 1.95 k +0.00%
hot/primitive/pass_uchar 2.758 k 2.75 k -0.29%
hot/primitive/pass_uint 1.979 k 1.979 k +0.00%
hot/primitive/pass_ullong 2.047 k 2.047 k +0.00%
hot/primitive/pass_ulong 1.979 k 1.979 k +0.00%
hot/primitive/pass_ushort 1.954 k 1.954 k +0.00%
hot/primitive/pass_wchar 1.98 k 1.98 k +0.00%
hot/string/pass_cstring 2.228 k 2.228 k +0.00%
hot/string/pass_string 20.41 k 20.38 k -0.12%
hot/string/pass_string_cref 20.04 k 20.07 k +0.14%
hot/string/pass_string_view 10.44 k 10.43 k -0.14%
hot/template/tmpl_deduce(bool) 2.18 k 2.18 k +0.00%
hot/template/tmpl_deduce(float) 2.186 k 2.186 k +0.00%
hot/template/tmpl_deduce(int) 2.12 k 2.12 k +0.00%
hot/template/tmpl_pass[bool] 6.288 k 6.134 k -2.45%
hot/template/tmpl_pass[char] 6.816 k 6.811 k -0.07%
hot/template/tmpl_pass[double] 6.034 k 6.053 k +0.31%
hot/template/tmpl_pass[float] 6.192 k 6.203 k +0.18%
hot/template/tmpl_pass[long double] 6.364 k 6.191 k -2.72%
hot/template/tmpl_pass[long] 6.16 k 6.349 k +3.07%
hot/template/tmpl_pass[short] 6.192 k 6.211 k +0.31%
hot/template/tmpl_pass[unsigned int] 6.103 k 6.285 k +2.98%
hot/template/tmpl_pass[unsigned long long] 6.549 k 6.546 k -0.05%
hot/template_overload/tmpl_overloaded_1(Particle) 2.875 k 2.875 k +0.00%
hot/template_overload/tmpl_overloaded_1(Point) 2.853 k 2.853 k +0.00%
hot/template_overload/tmpl_overloaded_1(Widget) 2.865 k 2.865 k +0.00%
hot/template_overload/tmpl_overloaded_2(Particle) 2.434 k 2.434 k +0.00%
hot/template_overload/tmpl_overloaded_2(Point) 2.414 k 2.414 k +0.00%
hot/template_overload/tmpl_overloaded_2(Widget) 2.424 k 2.424 k +0.00%
hot/template_overload/tmpl_overloaded_3(Particle,Point) 3.89 k 3.89 k +0.00%
hot/template_overload/tmpl_overloaded_3(Point,Widget) 3.885 k 3.885 k +0.00%

benchmarks/ measures what calling C++ functions through cppjit costs,
as the instructions callgrind counts for a call. Counts do not depend
on the machine's load, so one call per case is enough, and runs on
different machines with the same toolchain compare.

The C++ side (benchmarks/cpp/, built by benchmarks/Makefile into
callsDict.so) declares the benchmarked API once and includes it into
three namespaces: `hot` for successive calls, and `warmup` and `cold`
for first calls, since a function has only one first call. The API
covers single-overload functions over every primitive type (by value,
pointer, reference and const reference), strings, classes and enums; a
non-templated overload set; single and overloaded templates; and
overloaded higher order functions taking a Python callback. The bodies
only return their input, so the counts are cppjit's overhead. Markers
in the library toggle callgrind's counting and dump each count under
its test's name.

test_calls.py measures every case twice: the first call, from `cold`
after warming up on `warmup`, and a successive call, from `hot`. Only
the call is counted, with the garbage collector held off; the lookup
and the arguments are built before. Every result is checked, so a case
that resolves to the wrong overload fails instead of measuring the
wrong call. Run without callgrind, the tests only check.

callgrind.sh counts the suite. It rebuilds the library with the
valgrind headers, and runs Python with ASLR off, a fixed hash seed and
a fixed environment, since pointer-keyed hash tables and the
environment's size otherwise move the counts from run to run.

report.py collects the dumps into JSON, compares two sets of counts
into a markdown report that fails above a threshold, and exports
counts for github-action-benchmark.
A new Benchmark workflow counts benchmarks/ under callgrind on Linux
x86_64, with Release builds against the pinned CppInterOp.

On a push to main a separate job, the only one with write access,
publishes the counts to gh-pages: github-action-benchmark charts every
benchmark (first call and successive calls) over time, and
benchmarks/baseline.json keeps the latest counts for pull requests.

A pull request builds and counts only itself, then compares against
that baseline. Counts depend on the toolchain, so each carries a
fingerprint (OS image, glibc, g++, LLVM, Python, valgrind). main is
built and counted on the same runner when the baseline is missing,
comes from another fingerprint, or shows a regression, which this then
confirms. The job fails when a benchmark runs more than 5% more
instructions than main. A report job posts the comparison as a PR
comment, updated in place like the test results comment.

The lint workflow also checks the formatting of the benchmarks' C++
sources.
changelog/2026-10-21-Benchmarks.md describes the benchmarks and their
CI: what they measure, how to run them, how pull requests are checked
against main, and their known limitations.
@Vipul-Cariappa
Vipul-Cariappa marked this pull request as ready for review October 21, 2026 14:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant